Tracking objects on a touch surface

ABSTRACT

A device implements a method of tracking objects on a touch surface of an FTIR based touch-sensitive apparatus. An interaction map is generated that indicates local changes in interaction on the touch surface. The interaction map is processed for identification of positive and negative peaks that represent a locally increased and decreased interaction in the interaction map, respectively. To suppress the impact of interferences, a dedicated first heuristic and/or second heuristic is applied to identify potentially false peaks among the positive peaks. The first heuristic designates a positive peak as a potentially false peak when the positive peak is deemed to be associated with one or more of the negative peaks. The second heuristic designates a positive peak as a potentially false peak when the positive peak is deemed to be located along one of the movement trajectories. Movement trajectories of all objects deemed to exist on the touch surface at a preceding time point are then updated while taking the potentially false peaks into account.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of, and claims priority under35 U.S.C. § 120 to, U.S. patent application Ser. No. 14/365,468, filedon 13 Jun. 2014, which is the National Stage under 35 U.S.C. § 371 ofPCT International Application No. PCT/SE2012/051368, filed 10 Dec. 2012,which claims priority to Swedish patent application No. 1151208-4, filed16 Dec. 2011, and to provisional U.S. patent application No. 61/576,550,filed 16 Dec. 2011, the entire contents of all of which are incorporatedherein by reference.

TECHNICAL FIELD

The present invention relates to techniques for tracking objects on atouch surface, in particular a touch surface of a touch-sensitiveapparatus that operates by propagating light inside a light transmissivepanel from a plurality of incoupling points to a plurality ofoutcoupling points so as to illuminate the touch surface from within thepanel.

BACKGROUND ART

Touch sensing systems (“touch systems”) are in widespread use in avariety of applications. Typically, the touch systems are actuated by atouch object such as a finger or stylus, either in direct contact, orthrough proximity (i.e. without contact), with a touch surface. Touchsystems are for example used as touch pads of laptop computers, incontrol panels, and as overlays to displays on e.g. hand held devices,such as mobile telephones. A touch panel that is overlaid on orintegrated in a display is also denoted a “touch screen”. Many otherapplications are known in the art.

To an increasing extent, touch systems are designed to be able to detecttwo or more touches simultaneously, this capability often being referredto as “multi-touch” in the art.

There are numerous known techniques for providing multi-touchsensitivity, e.g. by using cameras to capture light scattered off thepoint(s) of touch on a touch panel, or by incorporating resistive wiregrids, capacitive sensors, strain gauges, etc into a touch panel.

WO2011/028169 and WO2011/049512 disclose multi-touch systems that arebased on frustrated total internal reflection (FTIR). Light sheets arecoupled into a panel to propagate inside the panel by total internalreflection (TIR). When an object comes into contact with a touch surfaceof the panel, the propagating light is attenuated at the point of touch.The transmitted light is measured at a plurality of outcoupling pointsby one or more light sensors. The signals from the light sensors areprocessed for input into an image reconstruction algorithm thatgenerates a 2D representation of interaction across the touch surface.This enables repeated determination of current position/size/-shape oftouches in the 2D representation while one or more users interact withthe touch surface.

The touches need to be detected in the presence of interferences, suchas measurement noise, ambient light, contamination on the touch surface(e.g. fingerprints and other types of smear), vibrations, etc. Theseinterferences may vary not only over time but also across the touchsurface, making it difficult to properly detect the touches on the touchsurface at all times. Furthermore, the degree of interaction between atouching object and the touch surface may vary both over time andbetween different objects. For example, the interaction may depend on ifan object is tapped, dragged or held in a fixed position onto the touchsurface. Different objects may yield different degree of interaction,e.g. the degree of interaction may vary between fingers of a user, andeven more so between the fingers of different users.

The combination of several touches, complex gestures as well as temporaland spatial variations of interaction and interferences will make theidentification of touches a demanding task. The user experience may behampered if, e.g., an ongoing gesture on a touch screen is interruptedby the system failing to detect certain touches during the gesture.

The above-mentioned WO2011/028169 and WO2011/049512 propose differenttechniques to suppress the influence of contamination, e.g. bycompensating the 2D representation by an intermittently updated 2Dbackground status that estimates the interaction caused bycontamination, or by normalizing the measurement signals by referencevalues that are intermittently updated to represent the influence ofcontamination. To the extent that such a suppression is incomplete, orother types of interferences are present, the 2D representation maystill contain false peaks that are mistaken for touching objects.

The prior art also comprises WO2010/006883 which discloses a techniquefor touch determination in an optical multi-touch system, whereattenuated light paths are identified and processed for determining aset of candidate touches, which may include true touches as well asghost touches. The candidate positions are then validated based theshape and/or area of each candidate touch, allowing the ghost touches tobe identified and eliminated from further processing.

The present Applicant has identified a need for an improved technique oftracking objects on the touch surface in the presence of interferences.

SUMMARY

It is an objective of the invention to at least partly overcome one ormore of the above-identified limitations of the prior art.

Another objective is to enable tracking of objects on a touch surface ofa multi-touch system that operates by FTIR.

A further objective is to enable such tracking based on a 2Drepresentation of interaction on a touch surface, even if the 2Drepresentation contains peaks caused by interferences.

One or more of these objectives, as well as further objectives that mayappear from the description below, are at least partly achieved by meansof a method, a computer program product, a device and a touch-sensitiveapparatus according to the independent claims, embodiments thereof beingdefined by the dependent claims.

A first aspect of the invention is a method of tracking movement of oneor more objects on a touch surface of a touch-sensitive apparatus, basedon an output signal of the apparatus. The apparatus is operable topropagate light by internal reflections inside a light transmissivepanel from a plurality of incoupling points to a plurality ofoutcoupling points so as to illuminate the touch surface from within thelight transmissive panel, wherein at least one light detector isoptically coupled to the outcoupling points to generate the outputsignal representing received light at the outcoupling points. The methodcomprises the steps of: processing the output signal to generate aninteraction map that indicates local changes in interaction on the touchsurface; processing the interaction map for identification of positiveand negative apparent peaks, wherein each positive apparent peakrepresents a locally increased interaction in the interaction map andeach negative apparent peak represents a locally decreased interactionin the interaction map; processing the positive and negative apparentpeaks to identify potentially false peaks among the positive andnegative apparent peaks; and updating, based on the potentially falsepeaks, movement trajectories of all objects deemed to exist on the touchsurface at a preceding time point; wherein the step of processing thepositive and negative apparent peaks applies at least one of a firstheuristic and a second heuristic for identifying the potentially falsepeaks, said first heuristic designating a positive apparent peak as apotentially false peak when the positive apparent peak is deemed to beassociated with one or more of the negative apparent peaks, and saidsecond heuristic designating a positive apparent peak as a potentiallyfalse peak when the positive apparent peak is deemed to be located alongone of the movement trajectories.

The method of the first aspect, when executed at consecutive timepoints, results in movement trajectories that are updated at each timepoint to represent all objects that are deemed to exist on the touchsurface at that time point. As used herein, a movement trajectoryrepresents a time sequence of locations for one object on the touchsurface. The method thus tracks the movement of objects on the touchsurface. Although not stated above, it is implicit to the method that itcomprises a step of processing the interaction map for identifying truepeaks among the positive and negative peaks, i.e. peaks that are safelydeemed to correspond to an object that is controlled by a user of thetouch-sensitive apparatus, and a step of updating the movementtrajectories based on the true peaks. This is the conventional way oftracking movement of objects of a touch surface.

The method of the first aspect validates the positive and negativeapparent peaks, to identify potentially false peaks in the interactionmap, and updates the movement trajectories based on the outcome of thevalidation. In contrast to true peaks, false peaks are peaks that do notcorrespond to an object that is controlled by the user. One challengewhen identifying the potentially false peaks is to reduce the number ofapparent peaks that are erroneously identified as new touches, whileensuring that those apparent peaks that indeed represent new touches arenot mistaken for false peaks. Errors introduced into the movementtrajectories may propagate, and even be enhanced, as the movementtrajectories are updated over time. To contain this risk, the method ofthe first aspect applies the first and/or second heuristic which makesuse of inherent characteristics of the touch-sensitive apparatus whenvalidating the apparent peaks. These inherent characteristics relate tohow true peaks are known to show up in the interaction map. For example,an existing object, moving or not, will show up as an apparent peak inthe interaction map at a location relatively close to its previouslocation, i.e. its location at the preceding time point. Furthermore,new objects that are brought into proximity with the touch surface willappear as positive peaks, i.e. peaks of locally increased interaction inthe interaction map.

The inherent characteristics also relate to how certain false peaks areknown to show up in the interaction map. Specifically, the first andsecond heuristics are designed to handle specific types ofinterferences. The first heuristic is designed to identify false peaksoriginating from light scattering caused by objects and contamination onthe touch surface. Such scattering has been found to cause aredistribution of energy at the outcoupling points which results in aninteraction map with one or more negative peaks in the vicinity of thetrue peak, i.e. the positive peak that corresponds to the object. Theredistribution of energy may also lead to one or more further positivepeaks in the vicinity of the true peak. The second heuristic is designedto identify false peaks originating from changes in the contamination onthe touch surface caused by the movement of the object(s). Such changesin contamination, also denoted “residue changes” herein, may be causedby the object leaving behind a trail of contamination and/or wiping awayexisting contamination on the touch surface. Such residue changes mayresult in one or more positive peaks along the movement trajectory ofthe object.

Depending on the impact of different types of interferences in aspecific touch-sensing apparatus, the first and/or second heuristic maybe applied to identify potentially false peaks among the apparent peaks,while still being able to identify apparent peaks representing newtouches.

It should also be noted that the first and/or second heuristic may beable to identify false peaks originating from other types ofinterferences, including but not limited to incomplete contaminationcompensation, ambient light and measurement noise.

In one embodiment, the first heuristic is designed to further requiresaid one or more of the negative apparent peaks to be located outsidethe movement trajectories. Such an embodiment may e.g. be applicablewhen the interaction map is known to represent objects by negativeapparent peaks, and it is thus conceivable that negative apparent peaksoccur along the movement trajectories. To improve the accuracy of thepotentially false peaks that are identified, the first heuristic may bedesigned to exclude such negative apparent peaks.

In one embodiment, with respect to the first heuristic, the positiveapparent peak is deemed to be associated with said one or more negativeapparent peaks when the positive apparent peak has a given spatialrelation to said one or more negative apparent peaks. The given spatialrelation may define a maximum distance between the positive apparentpeak and said one or more negative peaks. In an alternative, the givenspatial relation may be represented by one or more predeterminedpatterns of positive and negative apparent peaks in the interaction map.

In one embodiment, the step of processing the negative and positiveapparent peaks comprises a step of matching at least part of thepositive and negative apparent peaks to the movement trajectories, so asto identify a set of matched apparent peaks and a set of non-matchedapparent peaks.

In one embodiment, the second heuristic further requires, to designatethe positive apparent peak as a potentially false peak, that thepositive apparent peak is included in said set of non-matched apparentpeaks. This may prevent a true peak representing an object that crossesan existing movement trajectory from being identified as a potentiallyfalse touch.

In one embodiment, the second heuristic further requires, to designatethe positive apparent peak as a potentially false peak, that thepositive apparent peak is located within a maximum distance from acurrent location of said one of the movement trajectories. This mayprevent a new touch that occurs in an existing movement trajectory frombeing identified as a potentially false touch. In one embodiment, themaximum distance is a distance along said one of the movementtrajectories, and the second heuristic is configured to dynamically setthe maximum distance to increase with increasing velocity of said one ofthe movement trajectories. This embodiment is based on the insight thatfast moving objects quickly leave the residue changes behind on thetouch surface, and that these residue changes may be mistaken for newtouches unless care is taken when setting the maximum distance. Bysetting the maximum distance as a function of velocity (in contrast toan absolute distance), it is ensured that only the most recent part ofthe movement trajectory is evaluated by the second heuristic whenidentifying the potentially false peaks.

In one embodiment, the step of updating comprises: evaluating, based onthe potentially false peaks, the positive and negative apparent peaksfor determination of a current location for at least part of themovement trajectories; evaluating, based on the potentially false peaks,the positive and negative apparent peaks for identification of newtouches; and adding the location of each of said new touches as astarting point of a new movement trajectory. In one embodiment, the stepof updating comprises: preventing the identification of new touchesbased on the potentially false peaks. In one embodiment, the methodoperates in successive repetitions of the steps of processing andupdating, and the identification of new touches based on the potentiallyfalse peaks is prevented during a number of successive repetitions forat least part of the potentially false peaks, thereby postponing theidentification of new touches.

In one embodiment, the step of updating comprises at least one of:updating the movement trajectories based on said at least part of thepositive and negative apparent peaks while omitting all potentiallyfalse peaks; updating the movement trajectories based on said at leastpart of the positive and negative apparent peaks while applying arespective peak probability value indicative of the potentially falsepeaks; postponing the updating of at least a subset of movementtrajectories that are deemed to be associated with the potentially falsepeaks; creating tentative movement trajectories for at least a subset ofthe potentially false peaks; postponing the updating of movementtrajectories that are deemed to be associated with one or moreuncertainty regions containing the potentially false peaks in theinteraction map.

The step of processing the apparent positive and negative apparentpeaks, i.e. the validation, may be seen to generate validation data insome form, and the step of updating the movement trajectories may beseen to operate on this validation data. The step of updating may differdepending on validation data.

In a first example, the validation data designates a set of the apparentpeaks as either false or true. In a second example, the validation dataassigns probability values to the apparent peaks indicating a likelihoodthat the respective apparent peak is false (or equivalently, true). Thismay enable more sophisticated updating, e.g. to account foruncertainties in the identification of the potentially false peaks. In athird example, the validation data indicates one or more uncertaintyregions in the interaction map. The extent of the respective uncertaintyregion may be fixed or be adjusted with respect to the extent andrelative locations of certain apparent peaks and/or movementtrajectories.

In one embodiment, the step of processing the output signal comprises:processing the output signal to identify signal values for a pluralityof light paths extending between pairs of incoupling and outcouplingpoints, and generating the interaction map by operating an imagereconstruction algorithm on the signal values. The interaction map maybe generated to represent changes in interaction on any time scale. Inone example, the interaction map may be generated to represent thefactual touch interaction across the touch surface at the current timepoint. Such an interaction map will represent changes in interaction ona long time scale, e.g. all interaction changes since startup orcalibration of the touch-sensitive apparatus. This interaction map maythus represent an instant “snapshot” of the distribution of interactionacross the touch surface. In another example, the interaction map isgenerated to indicate temporal changes in interaction on a shorter timescale, e.g. in the approximate range of 5 ms-5 s. Irrespective ofimplementation, the interaction map may or may not be compensated forthe influence of contamination on the touch surface, e.g. according tothe prior art techniques discussed in the Background section.

In one embodiment, the interaction is represented as one of anattenuation and a transmission of said light. Thus, depending onimplementation, a positive apparent peak, which presents a locallyincreased interaction, may actually be represented by a local maximum orminimum in the interaction map.

In one embodiment, the method further comprises a step of identifyingcurrent touch data for at least part of the updated movementtrajectories, and outputting the touch data. The touch data may thusrepresent all true touches, or a subset of the true touches. The touchdata may comprise any combination of a location on the touch surface, asize, a shape and an application pressure onto the touch surface.

In one embodiment, the method further comprises: retrieving, from anelectronic memory, a data structure indicative of the movementtrajectories of all objects deemed to exist on the touch surface at thepreceding time point, wherein the step of updating comprises updatingthe movement trajectories in the data structure, and storing the updateddata structure in the electronic memory.

A second aspect of the invention is a computer program productcomprising computer code which, when executed on a data-processingsystem, is adapted to carry out the method of the first aspect.

A third aspect of the invention is a device for tracking movement of oneor more objects on a touch surface of a touch-sensitive apparatus. Thetouch-sensitive apparatus is operable to propagate light by internalreflections inside a light transmissive panel from a plurality ofincoupling points to a plurality of outcoupling points so as toilluminate the touch surface from within the light transmissive panel,wherein a light sensing means is optically coupled to the outcouplingpoints to generate an output signal representing received light at theoutcoupling points. The device comprises: means for processing theoutput signal to generate an interaction map that indicates localchanges in interaction on the touch surface; means for processing theinteraction map for identification of positive and negative apparentpeaks, wherein each positive apparent peak represents a locallyincreased interaction in the interaction map and each negative apparentpeak represents a locally decreased interaction in the interaction map;means for processing the positive and negative apparent peaks toidentify potentially false peaks among the positive and negativeapparent peaks; and means for updating, based on the potentially falsepeaks, movement trajectories of all objects deemed to exist on the touchsurface at a preceding time point; wherein the means for processing thepositive and negative apparent peaks is configured to apply at least oneof a first heuristic and a second heuristic for identifying thepotentially false peaks, said first heuristic designating a positiveapparent peak as a potentially false peak when the positive apparentpeak is deemed to be associated with one or more of the negativeapparent peaks, and said second heuristic designating a positiveapparent peak as a potentially false peak when the positive apparentpeak is deemed to be located along one of the movement trajectories.

A fourth aspect of the invention is a device for tracking movement ofone or more objects on a touch surface of a touch-sensitive apparatus.The touch-sensitive apparatus is operable to propagate light by internalreflections inside a light transmissive panel from a plurality ofincoupling points to a plurality of outcoupling points so as toilluminate the touch surface from within the light transmissive panel,wherein at least one light detector is optically coupled to theoutcoupling points to generate an output signal representing receivedlight at the outcoupling points. The device comprises: an input for theoutput signal; and a signal processor configured to: process the outputsignal to generate an interaction map that indicates local changes ininteraction on the touch surface; process the interaction map foridentification of positive and negative apparent peaks, wherein eachpositive apparent peak represents a locally increased interaction in theinteraction map and each negative apparent peak represents a locallydecreased interaction in the interaction map; process the positive andnegative apparent peaks to identify potentially false peaks among thepositive and negative apparent peaks; and update, based on thepotentially false peaks, movement trajectories of all objects deemed toexist on the touch surface at a preceding time point; wherein the signalprocessor is further configured to apply at least one of a firstheuristic and a second heuristic for identifying the potentially falsepeaks, said first heuristic designating a positive apparent peak as apotentially false peak when the positive apparent peak is deemed to beassociated with one or more of the negative apparent peaks, and saidsecond heuristic designating a positive apparent peak as a potentiallyfalse peak when the positive apparent peak is deemed to be located alongone of the movement trajectories.

A fifth aspect of the invention is a touch-sensitive apparatus,comprising: a light transmissive panel configured to propagate light byinternal reflections from a plurality of incoupling points to aplurality of outcoupling points so as to illuminate the touch surfacefrom within the light transmissive panel; means for providing the lightat the incoupling points; means for generating an output signalrepresenting received light at the outcoupling points; and the devicefor tracking movement of one or more objects according to the third orfourth aspect.

A sixth aspect of the invention is a touch-sensitive apparatus,comprising: a light transmissive panel configured to propagate light byinternal reflections from a plurality of incoupling points to aplurality of outcoupling points so as to illuminate the touch surfacefrom within the light transmissive panel; at least one light emitteroptically coupled to the incoupling points to provide the light; atleast one light detector optically coupled to the outcoupling points togenerate an output signal representing received light at the outcouplingpoints; and the device for tracking movement of one or more objectsaccording to the third or fourth aspect.

Any one of the above-identified embodiments of the first aspect may beadapted and implemented as an embodiment of the second to sixth aspects.

Still other objectives, features, aspects and advantages of the presentinvention will appear from the following detailed description, from theattached claims as well as from the drawings.

BRIEF DESCRIPTION OF DRAWINGS

Embodiments of the invention will now be described in more detail withreference to the accompanying schematic drawings.

FIGS. 1A-1B are section and top plan views of a touch-sensing FTIRsystem.

FIGS. 2A-2C are 3D plots of a total attenuation pattern, a backgroundpattern, and an offset pattern, respectively.

FIGS. 3A-3B are plots of a difference pattern representing two movingobjects.

FIG. 4 illustrate movement trajectories of two objects on a touchsurface.

FIG. 5A is a flow chart of a method for tracking objects according to anembodiment of the invention, and FIG. 5B is a block diagram of astructure for implementing the method in FIG. 5A.

FIG. 6 illustrates a process of maintaining a trace history list whichis used in the method of FIG. 5A.

FIGS. 7A-7D are plan views of an FTIR system to illustrate howinterferences may be caused by light scattering.

FIG. 8 is a plot of an offset pattern containing a true peak and falsepeaks caused by light scattering.

FIGS. 9A-9B are plan views of an FTIR system to illustrate howinterferences may be caused by changes in contamination.

FIG. 10 is a plot of an offset pattern containing a true peak and falsepeaks caused by changes in contamination.

FIG. 11 is a plot of a difference pattern containing true peaks andfalse peaks caused by changes in contamination.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

Before describing embodiments of the invention, a few definitions willbe given.

A “touch object” or “touching object” is a physical object that activelyor intentionally touches, or is brought in sufficient proximity to, atouch surface so as to be detected by one or more sensors in the touchsystem. The physical object may be animate, e.g. a finger, or inanimate.

An “interaction” occurs when the touch object affects a parametermeasured by the sensor.

An “interaction strength” is a relative or absolute measure of thedegree of interaction.

An “interaction map” or “interaction pattern” is a two-dimensional (2D)distribution of interaction across the touch surface, or a portionthereof. If the “interaction map” contains attenuation values, it isalso denoted an “attenuation map” or “attenuation pattern”.

A “touch” denotes a point of interaction as seen in the interactionpattern. A touch may be associated with different touch parameters, suchas a location on the touch surface in any coordinate system, aninteraction strength, a size (e.g. diameter, area, etc), a shape, etc.

A “frame” or “iteration” denotes a repeated event starting with datacollection and ending with determination of touch data. The touch datamay comprise or be based on any combination of touch parameters.

As used herein, a “trace” is information about the temporal history ofan interaction. Touches from an interaction detected in a sequence offrames, i.e. at different points in time, are collected into a trace.

A trace may be associated with different trace parameters, such as anage, a location, a location history, a size, a shape, etc. The “age” ofa trace indicates how long the trace has existed, and may be given as anumber of frames, the frame number of the earliest touch in the trace, atime period, etc. The “location” of the trace is given by the locationof the most recent touch in the trace, and the “size” of a trace denotesthe size of the most recent touch in the trace. The “location history”or “movement trajectory” denotes at least part of the spatial extensionof the trace across the touch surface, e.g. given as the locations ofthe latest few touches in the trace, or the locations of all touches inthe trace, a curve approximating the shape of the trace, or a Kalmanfilter.

Throughout the following description, the same reference numerals areused to identify corresponding elements.

1. Touch-Sensitive Apparatus

FIGS. 1A-1B illustrate an example embodiment of a touch-sensitiveapparatus 100 that is based on the concept of FTIR (Frustrated TotalInternal Reflection), also denoted “FTIR system”. The apparatus operatesby transmitting light inside a panel 1, from light emitters 2 to lightsensors or detectors 3, so as to illuminate a touch surface 4 fromwithin the panel 1. The panel 1 is made of solid material in one or morelayers and may have any shape. The panel 1 defines an internal radiationpropagation channel, in which light propagates by internal reflections.In the example of FIG. 1, the propagation channel is defined between theboundary surfaces 5, 6 of the panel 1, where the top surface 5 allowsthe propagating light to interact with touching objects 7 and therebydefines the touch surface 4. This is achieved by injecting the lightinto the panel 1 such that the light is reflected by total internalreflection (TIR) in the touch surface 4 as it propagates through thepanel 1. The light may be reflected by TIR in the bottom surface 6 oragainst a reflective coating thereon. It is also conceivable that thepropagation channel is spaced from the bottom surface 6, e.g. if thepanel comprises multiple layers of different materials. The apparatus100 may be designed to be overlaid on or integrated into a displaydevice or monitor.

The apparatus 100 allows an object 7 that is brought into close vicinityof, or in contact with, the touch surface 4 to interact with thepropagating light at the point of touch. In this interaction, part ofthe light may be scattered by the object 7, part of the light may beabsorbed by the object 7, and part of the light may continue topropagate in its original direction across the panel 1. Thus, thetouching object 7 causes a local frustration of the total internalreflection, which leads to a decrease in the energy (power/-intensity)of the transmitted light, as indicated by the thinned lines downstreamof the touching objects 7 in FIG. 1A.

The emitters 2 are distributed along the perimeter of the touch surface4 to generate a corresponding number of light sheets inside the panel 1.In the example of FIG. 1B, each emitter 2 generates a beam of light thatexpands in the plane of the panel 1 while propagating in the panel 1.Each beam propagates from one or more entry or incoupling points (ports)on the panel 1. The sensors 3 are distributed along the perimeter of thetouch surface 4 to receive the light from the emitters 2 at a number ofspaced-apart outcoupling points (ports) on the panel 1. It should beunderstood that the incoupling and outcoupling points merely refer tothe position where the beam enters and leaves, respectively, the panel1. Thus, one emitter/sensor may be optically coupled to a number ofincoupling/outcoupling points. In the example of FIG. 1B, however, thedetection lines D are defined by individual emitter-sensor pairs.

The sensors 3 collectively provide an output signal, which is receivedand sampled by a signal processor 10. The output signal contains anumber of sub-signals, also denoted “projection signals”, eachrepresenting the energy of light emitted by a certain light emitter 2and received by a certain light sensor 3. Depending on implementation,the signal processor 10 may need to process the output signal forseparation of the individual projection signals. Conceptually, the touchapparatus 100 is considered to define a grid of detection lines D on thetouch surface 4, where each detection line D corresponds to a lightpropagation path from an emitter 2 to a sensor 3, as projected onto thetouch surface 4. Thus, the projection signals represent the receivedenergy or power of light on the individual detection lines D. It isrealized that the touching object 7 results in a decrease (attenuation)of the received energy on one or more detection lines D.

As will be explained below, the signal processor 10 may be configured toprocess the projection signals so as to determine a distribution ofattenuation values (for simplicity, referred to as an “attenuationpattern”) across the touch surface 1, where each attenuation valuerepresents a local attenuation of light. The attenuation pattern may berepresented in many different ways, e.g. as attenuation values arrangedin a regular x-y-grid, such as in an ordinary digital image, althoughother types of grids are conceivable, e.g. hexagonal patterns ortriangular meshes. The attenuation pattern may be further processed bythe signal processor 10 or by a separate device (not shown) for touchdetermination, which may involve extraction of touch data, such as aposition (e.g. x,y coordinates), a shape or an area of each touchingobject.

In the illustrated example, the apparatus 100 also includes a controller12 which is connected to selectively control the activation of theemitters 2 and, possibly, the readout of data from the sensors 3. Thesignal processor 10 and the controller 12 may be configured as separateunits, or they may be incorporated in a single unit. One or both of thesignal processor 10 and the controller 12 may be at least partiallyimplemented by software executed by a processing unit 14.

It is to be understood that FIG. 1 merely illustrates one example of anFTIR system. For example, the detection lines may instead be generatedby sweeping or scanning one or more beams of light inside the panel 1,and the light may instead be coupled into and out of the panel 1 via thetop and bottom surfaces 5, 6, e.g. by the use of dedicated couplingelements attached to the panel 1. Examples of alternative FTIR systemsare e.g. disclosed in U.S. Pat. No. 6,972,753, U.S. Pat. No. 7,432,893,US2006/0114237, US2007/0075648, WO2009/048365, WO2010/006882,WO2010/006883, WO2010/006884, WO2010/006885, WO2010/006886,WO2010/064983, and WO2010/134865, which are all incorporated herein bythis reference. The inventive concept may be applied to such alternativeFTIR systems as well.

2. Attenuation Pattern

A reconstruction function or algorithm may be used for determining theattenuation pattern on the touch surface 4, based on the projectionsignals in the output signal. Embodiments of the invention may use anyavailable algorithm for image reconstruction based on projection signalvalues, including tomographic reconstruction methods such as FilteredBack Projection, FFT-based algorithms, ART (Algebraic ReconstructionTechnique), SART (Simultaneous Algebraic Reconstruction Technique), etc.Alternatively, the reconstruction function may generate the attenuationpattern by adapting one or more basis functions and/or by statisticalmethods such as Bayesian inversion. Examples of such reconstructionfunctions designed for use in touch determination are found in WO2010/006883, WO2009/077962, WO2011/049511, WO2011/139213 andWO2012/050510, all of which are incorporated herein by reference.Conventional image reconstruction techniques are found in themathematical literature, e.g. “The Mathematics of ComputerizedTomography” by Natterer, and “Principles of Computerized TomographicImaging” by Kak and Slaney.

The reconstruction function is designed based on the assumption that theinput values s_(t) at time point t depend on the attenuation pattern,a_(t), according to a projection function

, which reflects the properties of the physical touch system: s_(t)=

(a_(t)). The reconstruction algorithm is thus designed to reconstructa_(t) from s_(t) by use of a reconstruction function: a_(t)=

′(s_(t)).

It is to be understood that the format of the input values s_(t) may bespecific to the reconstruction function

′. To enable reconstruction of an attenuation pattern, the input valuess_(t) may be represented as attenuation values for individual detectionlines. Such attenuation values may e.g. be based on transmission valuesobtained by dividing the projection value I_(k) of the k:th detectionline by a respective reference value: T_(k)=I_(k)/REF_(k). By properchoice of reference values, the transmission values may be generated torepresent the fraction (e.g. in the range [0, 1]) of the available lightenergy that has been measured on each of the detection lines. The inputvalue of each detection line may e.g. be given by: s_(k)=−log(T_(k)) oran approximation thereof, e.g. s_(k)=1−T_(k).

The reconstructed attenuation pattern a_(t) will represent thedistribution of accumulated attenuation on the touch surface at timepoint t. The accumulated attenuation includes both attenuation caused bytouching objects and attenuation caused by contaminations on the touchsurface, and such an interaction pattern a_(t) may thus be denoted a“total attenuation pattern” or “total pattern”.

As indicated in the Background section, it is known in the art to atleast partly compensate the total attenuation pattern a_(t) for theinfluence of the contaminations on the touch surface, resulting in acompensated attenuation pattern o_(t), denoted “touch attenuationpattern” or “offset pattern”. For example, WO2011/049512 proposes atouch apparatus which repeatedly generates a current light status(corresponding to the total pattern). The touch apparatus alsorepeatedly updates a background status (or “background pattern”), whichis a two-dimensional distribution of estimated attenuation values causedby contaminations on the touch surface. A compensated light status(corresponding to the offset pattern) is then formed by subtracting thebackground status from the current light status. An alternative orsupplementary compensation technique is proposed in WO2011/028169, inwhich the reference values (REF_(k)) used in converting the projectionvalues into input values, are intermittently updated so as to includethe influence of contaminations on the touch surface 4. By tracking theinfluence of contaminations via the updating of the reference values,the touch apparatus compensates for the contaminations already in theinput to the reconstruction function, which thereby directly generatesthe offset pattern. WO2011/049512 and WO2011/028169, as well asApplicant's PCT application WO2013/048312, are all incorporated byreference in their entirety.

A total pattern a_(t) is illustrated in FIG. 2A as a 3D plot in thecoordinate system X,Y of the touch surface (cf. FIG. 1B). The patterna_(t) represents the accumulated attenuation on the touch surface, bothfrom touches and contamination. FIG. 2B is a plot of a backgroundpattern b_(t-1) which exhibits a first attenuation component α1 causedby a fingerprint from an earlier touch, a second attenuation componentα2 caused by smear from a palm resting on the touch surface, and a thirdattenuation component α3 caused by a liquid spill. FIG. 2C is a plot ofthe offset pattern o_(t) obtained by subtracting b_(t-1) from a_(t),pixel by pixel. Three peaks β1, β2, β3 forming three touches are seenagainst a uniform background level close to zero attenuation.

It is realized that this offset pattern o_(t) reflects long-term changesin touch-related attenuation on the touch surface, e.g. attenuationchanges compared to a clean touch surface without touching objects.

It is also possible to generate a differential attenuation pattern thatrepresents changes on a shorter time scale, typically changes that occurover a time period Δt which is (much) shorter than the temporal extentof traces. Such a differential attenuation patterns is denoted“difference pattern” herein and designated by da_(t).

The difference pattern da_(t) may be calculated by subtracting a currentoffset pattern with a previous one: da_(t)=o_(t)−o_(t-Δt).Alternatively, the difference pattern da_(t) may be calculated bysubtracting a current total pattern with a previous one:da_(t)=a_(t)−a_(t-Δt), provided that Δt is selected to be small enoughto avoid significant build up of contamination, while being long enoughfor the movement of touching objects to result in attenuation changes.In one example, the time difference is larger than about 5 ms,preferably about 20-80 ms but may be as large as 1-5 seconds.

In another alternative, the difference pattern da_(t) may bereconstructed from change values ds_(t) that are obtained by taking thedifference in input values between the current time point (t) and theprevious time point (t−Δt):

${{ds}_{t} = {{s_{t} - s_{t - {\Delta\; t}}} = {{{\log\left( \frac{I_{t - {\Delta\; t}}}{REF} \right)} - {\log\left( \frac{I_{t}}{REF} \right)}} = {{- {\log\left( {\frac{I_{t}}{REF}/\frac{I_{t - {\Delta\; t}}}{REF}} \right)}} = {- {\log\left( {I_{t}\text{/}I_{t - {\Delta\; t}}} \right)}}}}}},$

or, equivalently, by the difference in logarithmic signal values:ds _(t)=−log(I _(t) /I _(t-Δt))=log(I _(t-Δt))−log(I _(t)).

Thus, the reconstruction function

′ may operate on change values ds_(t) to generate the difference patternda_(t), assuming that the reconstruction function

′ is at least approximately linear:da _(t) =a _(t) −a _(t-Δt)=

′(s _(t))=

′(s _(t-Δt))=

′(s _(t) −s _(t-Δt))=

′(ds _(t)).

By operating on change values, the need to calculate, store and retrievethe total pattern or offset pattern at different time points may beobviated.

Further alternatives for generating the difference pattern are disclosedin Applicant's PCT application WO2013/055282, which is incorporated byreference in its entirety.

Irrespective of implementation, the difference pattern da_(t) is adistribution of difference values across the touch surface (or arelevant part of the touch surface), where each difference value mayrepresent a local change (increase/decrease) in attenuation during Δt ina specific position or reconstruction cell (pixel) on the touch surface.The difference pattern da_(t) may thus be seen as an image of the changein touch interaction and in contamination contribution during Δt acrossthe touch surface. The difference pattern da_(t) mainly indicates thetemporal change in attenuation caused by moving objects in contact withthe touch surface, as well objects that are removed from or added to thetouch surface during Δt. A moving object generally appears as a “dipolepeak” in the difference pattern, i.e. a combination of a positive peakand a negative peak. FIG. 3A is a 3D plot of a difference pattern da_(t)containing two dipole peaks γ₁ ⁺, γ₁ ⁻ and γ₂ ⁺, γ₂ ⁻. FIG. 3B is a planview of the difference pattern da_(t) in FIG. 3A. When an object ismoved during Δt, the corresponding touch occupies new pixels, resultingin a positive peak, and leaves old pixels, resulting in a negative peak.If an object is removed from the touch surface, the removed touch willbe represented by a single negative peak in the difference pattern, andif an object is brought into contact with the touch surface, the newtouch will be represented by a single positive peak. In FIG. 3B, it isrealized that the objects have moved away from each other during Δt.

Embodiments of the invention may operate on either the total pattern,the offset pattern or the difference pattern, or combinations of thesepatterns, to track movement of objects on the touch surface. In thefollowing, reference is therefore made to “attenuation pattern” or“pattern”, which is intended to designate all of the above-mentionedtypes of patterns, unless otherwise stated.

In a real-world implementation, the attenuation pattern will includeinterferences that make touch identification a demanding task. Theseinterferences may occur as peaks which may be mistaken for true touches.For example, interferences may be caused by measurement noise, artifactsintroduced by the reconstruction function, or ambient light, i.e. lightthat originates from sources outside of the panel and is detected by thesensors. Interferences may also originate from incomplete compensationfor contaminations on the touch surface. As will be explained furtherbelow, it is also possible that interferences are caused by the lightthat is scattered by the touching object.

3. Tracking

The touch apparatus 100 is operated to generate the pattern atconsecutive time points, so as to enable the movement of objects on thetouch surface to be tracked as a function of time. The tracking resultsin movement trajectories on the touch surface and may e.g. allow forrecognition of user-controlled gestures on the touch surface. As anexample, FIG. 4 illustrates movement trajectories 30 of two objects 7that are dragged on the touch surface 4 in a “pinch” gesture. It isrealized that the tracking is a demanding task, especially in thepresence of interferences, since it must not only correctly matchtouches detected at different times, but also correctly detect theoccurrence of new touches (i.e. a start of a new movement trajectory,“touch down”) and the disappearance of existing touches (i.e. atermination of an ongoing movement trajectory, “touch up”). It is alsoto be noted that touches may differ significantly in attenuation bothwithin the pattern and as a function of time. For example, movingtouches (“drags”) are often seen to yield weaker attenuation thanstationary touches in an FTIR-based touch apparatus.

FIG. 5A is a flow chart of an embodiment of a method for trackingobjects in the touch-sensitive apparatus 100 of FIG. 1. The methodinvolves a sequence of steps 50-57 that are executed repeatedly,typically by the signal processor 10 (FIG. 1). In the context of thisdescription, each sequence of steps 50-57 is denoted a “frame” or an“iteration”.

Each frame starts by a data collection step 50, in which the outputsignal is acquired from the light sensors 3, e.g. by sampling a valuefrom each of the aforesaid projection signals I_(t). The data collectionstep 50 results in one projection value for each detection line. It maybe noted that the data may, but need not, be collected for all availabledetection lines in the apparatus 100. The data collection step 50 mayalso include pre-processing of the projection values, e.g. filtering fornoise reduction.

In a conversion step 51, the projection values I_(t) are processed forconversion into input values s_(t) or change values ds_(t) depending onimplementation.

A subsequent pattern generation step 52 operates a reconstructionfunction

′ on the input or change values to generate the pattern, e.g. accordingto any of the techniques described in Chapter 2.

In a peak detection step 53, the pattern is then processed for detectionof peaks, e.g. using any known technique. In one embodiment, a global orlocal threshold is first applied to the pattern, to suppress noise. Anyareas with attenuation values that fall above the threshold may befurther processed to find local maxima. The identified maxima may befurther processed for determination of a touch shape and a centerposition, e.g. by fitting a two-dimensional second-order polynomial or aGaussian bell shape to the attenuation values, or by finding the ellipseof inertia of the attenuation values. Local minima may be identified insimilar way. There are also numerous other techniques as is well knownin the art, such as clustering algorithms, edge detection algorithms,standard blob detection, water shedding techniques, flood filltechniques, etc. Step 53 results in a peak list which may include valuesof one of the following peak parameters: peak position, peak size, peakshape, peak attenuation, and peak sign. The peak attenuation may begiven by a peak attenuation value or a weighted sum of attenuationvalues within the peak shape. The peak sign indicates if the peak ispositive or negative in the pattern.

A validation step 54 is designed to process the peak list foridentification of uncertain peaks (“possibly false peaks”), i.e. peaksthat are deemed likely (or deemed not unlikely) to originate frominterferences in a broad sense. Conceptually, step 54, may be separatedinto a matching step 54′ and a validation data generation step 54″.

The matching step 54′ matches the detected peaks in the peak list toexisting traces in a trace history list THL_(t-1), which is a record ofthe existing traces and is retrieved from electronic memory M. Therecord may be implemented by any suitable data structure, linear ornon-linear, such as list, a hash table, a binary tree, etc. The tracehistory list is maintained to contain values of trace parameters for thetraces deemed relevant to the current frame, e.g. all traces deemed toexist, which also may include any traces deemed not unlikely to exist.As indicated above, the trace parameters may e.g. include age, location,location history, attenuation strength, size, and shape for eachexisting trace. In FIG. 5A, the trace history lists updated in thelatest frame and in the current frame are indicated by THL_(t-1) andTHL_(t), respectively. Step 54′ results in a match list indicatingmatched peaks, non-matched peaks and non-matched traces.

The validation data generation step 54″ processes the peak list and/orthe match list for identification of uncertain peaks. To this end, step54″ applies rules that stipulate forbidden or allowed features of thedetected peaks. These rules typically make use of at least peak locationand peak sign to determine if a peak is potentially false or not. Thestep 54″ results in validation data υ_(t) which indicates the uncertainpeaks.

In an updating step 56, the trace history list THL_(t-1) is updatedbased on the match list and the validation data υ_(t). The updating stepis thus implemented to update relevant trace parameters of existingtraces, to remove terminated traces and to insert relevant traceparameters for new traces. The updated trace history list THL_(t) isstored in memory M for use by the matching step 54″ in a forthcomingframe.

Different embodiments and exemplifying combinations of steps 54 and 56as well as examples of validation data are given in Chapters 4 and 5below.

In step 57, current touch data is determined and output, whereupon theprocess returns to the data collection step 50. The current touch datamay be determined by selecting values determined during the peakdetection, matching or updating steps 53, 54′, 56, such as peakparameter values or trace parameter values. Step 57 may output touchdata for all existing and new traces as identified in step 56. However,it is conceivable that step 57 implements an “output filter”, whichmanipulates the current touch data to improve the user experience. Forexample, the output filter may postpone output of current touch data fornew traces for a number of frames to suppress the influence of erroneouspeak detections. Likewise, the output filter may be configured to delayremoval of a non-matched trace for a number of frames, e.g. byoutputting the latest touch data for the non-matched trace during theseframes or by outputting touch data that is projected for these framesbased on the trace parameter values in the touch history list. It isalso conceivable that the output filter is configured to make an activeselection among the traces found in step 56 and only output currenttouch data for the selected traces.

With reference to the embodiment in FIG. 5A, and variants thereof, it isto be understood that one or more of the indicated steps may be effectedconcurrently. For example, the data collection step 50 of a subsequentframes may be initiated concurrently with any one of the steps 51-57.

The above-described method may be implemented by a data processingdevice (cf. 10 in FIG. 1) which is connected to sample the measurementvalues from the light sensors 3 in the FTIR system. FIG. 5B shows anexample of such a data processing device 10, which includes an input102A for receiving the output signal. The device 10 further includes adata collection element (or means) 200 for obtaining the currentprojection values, a conversion element (or means) 201 for convertingthe current projection values into current input values or changevalues, a pattern generation element (or means) 202 for generating acurrent attenuation pattern, a peak detection element (or means) 203 fordetecting peaks in the current attenuation pattern, a validation element(or means) 204 for matching peaks to existing traces and for validatingpeaks according to the rules described herein, an updating element (ormeans) 205 for updating the trace history list, an outputting element(or means) 206 for providing current touch data for output, and anoutput 102B for outputting the current touch data.

The device 10 may be implemented by special-purpose software (orfirmware) run on one or more general-purpose or special-purposecomputing devices. In this context, it is to be understood that each“element” or “means” of such a computing device refers to a conceptualequivalent of a method step; there is not always a one-to-onecorrespondence between elements/means and particular pieces of hardwareor software routines. One piece of hardware sometimes implementsdifferent means/elements. For example, a processing unit may serve asone element/means when executing one instruction, but serve as anotherelement/means when executing another instruction. Naturally, it isconceivable that one or more elements (means) are implemented entirelyby analog hardware components.

The software controlled device 10 may include one or more processingunits (cf. 14 in FIG. 1B), e.g. a CPU (“Central Processing Unit”), a DSP(“Digital Signal Processor”), an ASIC (“Application-Specific IntegratedCircuit”), discrete analog and/or digital components, or some otherprogrammable logical device, such as an FPGA (“Field Programmable GateArray”). The device 10 may further include a system memory and a systembus that couples various system components including the system memoryto the processing unit. The system bus may be any of several types ofbus structures including a memory bus or memory controller, a peripheralbus, and a local bus using any of a variety of bus architectures. Thesystem memory may include computer storage media in the form of volatileand/or non-volatile memory such as read only memory (ROM), random accessmemory (RAM) and flash memory. The special-purpose software, the tracehistory list and any other data needed during execution, may be storedin the system memory, or on other removable/non-removablevolatile/non-volatile computer storage media which is included in oraccessible to the computing device, such as magnetic media, opticalmedia, flash memory cards, digital tape, solid state RAM, solid stateROM, etc. The data processing device 10 may include one or morecommunication interfaces, such as a serial interface, a parallelinterface, a USB interface, a wireless interface, a network adapter,etc, as well as one or more data acquisition devices, such as an A/Dconverter. The special-purpose software may be provided to the device 10on any suitable computer-readable medium, including a record medium, anda read-only memory.

4. Maintaining the Trace History List

The process of maintaining the trace history list according to steps 54and 56 will now be briefly explained with reference to FIG. 6, whileomitting the generation and use of validation data. The aim of such amaintaining process is to, based on identified matches between currentpeaks (detected in step 53) and traces in the trace history listTHL_(t-1), update the trace history list with current trace parametervalues. In FIG. 6, steps 61-62 belong to the matching step 54′ and steps63-65 belong to the updating step 56.

In step 61, the trace parameter values for the traces in the tracehistory list THL_(t-1) are processed for generating predicted values forthe current frame. Depending on implementation, step 61 may operate topredict the most likely location (coordinates), size and shape for alltraces in the trace history list. Step 61 may be omitted, but iscurrently believed to improve the quality of the distance measures whencomputing match values (below).

In step 62, the (predicted) values of the trace parameters are matchedagainst corresponding parameter values for the peaks in the peak list.Thereby, a match value is computed between every peak and every trace inthe trace history list. The match value indicates the likelihood of amatch between the peak and a specific trace.

The match value may, e.g., be computed as a function of the Euclidiandistance between the location of the peak and the (predicted) locationof the trace, and/or as a function of a similarity in size and/or shapeand/or attenuation strength. Also, invalid match values may be set forpeak-trace combinations that are deemed impossible. One limitation forpeak-trace combinations may be given by a maximum distance between the(predicted) location of the peak and the location of the trace. In oneexample, a match is deemed impossible if the distance is greater than 15mm. Imposing restrictions on allowed peak-trace combinations also makesit possible to use smarter algorithms for the match value computations,for example a hierarchical search over the touch surface.

It is to be understood that the match values may be computed to factorin further trace parameters included in the trace history list, e.g. theage of the trace.

In a total pattern and an offset pattern, a touching object isrepresented by a positive peak. For these types of patterns, step 62 mayignore all negative peaks, i.e. only positive peaks are matched againstexisting traces. In a difference pattern, a touching object may berepresented by both a positive and a negative peak (cf. FIG. 3). Fordifference patterns, step 62 may match either positive peaks or negativepeaks, or a combination thereof, against existing traces.

In step 62, the identified peaks are also assigned (i.e. matched) intothe trace history list based on the match values, so as to determine onetrace (if any) as a best match for each peak. A simple way to do thisassignment, if the match values are calculated to represent a bettermatch by a higher match value, is to start with the highest match valueand proceed to gradually lower match values (a greedy approach), andvice versa if a better match is represented by a lower match value.There are many other and more elaborate algorithms for making thisassignment, such as Hungarian method (Kuhn's algorithm) or any otheralgorithm solving bipartite matching based on match values. In analternative, the assignment accounts for the age of the trace associatedwith the respective match value, e.g. by starting with the oldest matchvalue/trace (largest value of age parameter) and proceeding to graduallyyounger match values/traces.

Step 62 results in the above-mentioned match list, in which each peak iseither assigned to a trace (“matched peak”) or is identified as lackingan assigned trace (“non-matched peak”). Likewise, each trace is eitherassigned to a peak or is identified as lacking an association(“non-matched trace”).

In step 63, the trace history list is updated for all matched peaks, byupdating the corresponding trace parameters as a function of the peakparameter values for the matched peak or peaks.

In step 64, non-matched peaks are interpreted as new traces and areadded to the trace history list Like in step 63, the trace parametervalues may be given by, or calculated as a function of, the peakparameter values.

In step 65, non-matched traces are interpreted as lost traces and areremoved from the trace history list. In one variant, the removal of sucha trace is delayed for a few (e.g. 1-10) frames to make it possible tofind peaks belonging to the trace in a later frame. This may e.g. avoidremoval of weak traces that may be temporarily hidden in noise orartifacts. In another variant, non-matched traces are removed if one ormore trace parameter values fulfils a given criterion, e.g. if the localnoise has decreased below what is expected, or if the attenuationstrength of a trace has decreased rapidly over a sequence of precedingframes (which may indicate that the corresponding object is being liftedfrom the touch surface).

5. Peak Validation and Use of Validation Data

As understood from FIG. 5A, step 54″ for generating validation data maybe executed between step 62 and steps 63-65 in FIG. 6, and steps 63-65may be configured to take into account the validation data generated bystep 54″.

Step 54″ applies a set of rules to find uncertain peaks among the peaks.The rules are tailored to find false peaks that are caused by phenomenaspecific to an FTIR system. Embodiments of the invention will now bedescribed with respect to two such phenomena: light scattering andcontamination (residue) changes.

5.1 False Peaks Caused by Light Scattering

Light scattering is a physical process that causes some of thepropagating light in the panel to deviate from its original path. In anFTIR system, there may be inherent scattering in the panel, e.g.scattering caused by irregularities in the boundary surfaces that definethe radiation propagation channel, and scattering caused byirregularities in the bulk material of the panel. This scattering may beinvariant over time and may be taken into account by calibration of theapparatus.

In an FTIR system, light scattering may also be caused by the touchingobjects, as well as contaminations on the touch surface, such asfingerprints and smear. This type of light scattering is dynamic, i.e.it changes as the system is used since both the location of the objectsand the distribution of contamination may vary over time. Furthermore,light scattering may differ depending on the type of object and how hardthe object is pressed against the touch surface. Likewise, lightscattering may differ for different types of contamination, e.g.fingerprints and liquid spills. For example, the pattern of valleys andridges on a human finger or in a fingerprint may cause irregularscattering of light, as well as interference/diffraction phenomenadepending on the coherence of the propagating light.

FIGS. 7A-7D illustrate the impact of light scattering on the projectionvalues measured by light sensors 3 in an FTIR system. For the purpose ofsimplification, the system is illustrated with one light emitter 2 andfive light sensors 3. In FIG. 7A, an object 7 is brought into contactwith the touch surface. In FIG. 7B, it is assumed that the apparatus isfree of dynamic light scattering. The emitter 2 is activated to generatea first and a second detection line. The first detection line passesdirectly through the contact area between the object 7 and the touchsurface, resulting in a decreased projection value (i.e. an increasedattenuation). The second detection line passes directly to the right ofthe contact area and will result in an unchanged projection value (i.e.no attenuation). FIG. 7C illustrate the effect of dynamic lightscattering. Some of the light that hits the contact area along isscattered by the object and impinges on the sensor of the seconddetection line. Thus, as shown in FIG. 7D, the projection value of thesecond detection line will represent the sum of the propagating lightreaching its sensor (FIG. 7B) and the scattered light reaching itssensor (FIG. 7C). This results in an increased projection value (i.e. adecreased attenuation) for the second detection line. The discussion inrelation to FIG. 7 is equally applicable when the object 7 is replacedby a local contamination, such as a fingerprint.

If the object 7 in FIG. 7D is now lifted from the touch surface (orequivalently, if the local contamination is at least partly removed),the projection value of the first detection line will increase while theprojection value of second detection line will decrease (since thescattered light is removed). Thus, the attenuation for the seconddetection line will increase, which may result in a positive peak in theresulting pattern that may be erroneously interpreted as a new touch.

It is realized that the dynamic light scattering results in are-distribution of light energy between detection lines. Typically, butnot generally, the dynamic light scattering affects detection lines thatpass close to the scattering object/contamination. This will lead to are-distribution of attenuation in the resulting pattern (generated instep 52 in FIG. 5A). Since no new light is created by the lightscattering, the sum of the re-distributed attenuation in the patternshould be zero. This means that dynamic light scattering results in bothpositive and negative peaks in the pattern. These scattering-inducedpeaks typically emanate as local phenomena around the scatteringobject/-contamination. FIG. 8 is a schematic illustration of a trace 30overlaid on an offset pattern (o_(t)), which contains peaks thatoriginate from a single touching object. The peaks form a confined groupwith two positive peaks and two negative peaks. One of the positivepeaks, which corresponds to the object, has been matched to the trace30, whereas the other three peaks are non-matched peaks. The validationstep 55 is designed to ensure that one or more of these non-matchedpeaks are not entered into the trace history list as new traces in theupdating step 56.

In one embodiment, step 54″ applies a “scatter heuristic” that includesa first rule that stipulates that a positive peak associated with (e.g.located in the vicinity of) one or more negative peaks is to bedesignated as an uncertain peak. In another variant, the first ruleapplies only to non-matched positive peaks, such that only one thenon-matched peak in FIG. 8 is designated as uncertain. The first rulemay be applied to positive peaks located, together with one or morenegative peaks, within a search region of given extent. The searchregion may, e.g. be defined around each negative peak. In a variant,indicated in FIG. 8, the search region R is defined around one or moreof the positive peaks, e.g. the positive peak that has been matched totrace 30. The extent of the search region R may be set in dependence ofthe design of the touch-sensitive apparatus, the reconstruction functionused in step 52 and the maximum conceivable size of the scatteringobject/contamination. In one non-limiting example, the search region Ris set to a diameter in the range of 1-50 mm. In an alternative, thepositive peaks associated with negative peaks are identified bysearching the peaks for specific spatial combinations of positive andnegative peaks, e.g. the quadropole of positive and negative peaks shownin FIG. 8. This “spatial pattern recognition” may also factor inattenuation strength, size, shape, etc of the individual peaks.

From FIG. 3B, it is realized that a difference pattern may presentnegative peaks that do not originate from interferences, but actuallyoccur along the movement trajectory (trace) of a touching object. Thus,when operating on difference patterns, the first rule applied in step54″ may, but need not, be modified to stipulate that positive peaks areto be designated as uncertain peaks only if the associated negative peak(or peaks) is (are) spatially separated from the current traces. Such amodified first rule may also be applied to offset patterns, which maypresent similar negative peaks, e.g. caused by delayed or incompletecontamination compensation.

In a further variant, the scatter heuristic may include a second rulethat all negative peaks are to be designated as uncertain peaks, so asto prevent negative peaks from be introduced into the trace historylist. However, the skilled person realizes that there are other ways ofexcluding negative peaks, e.g. to exclude all negative peaks in thematching step 54′ and/or to prevent the use of any negative peak in theupdating step 56.

Based on the scatter heuristic, step 54″ generates validation data thatindicates the uncertain peaks. The validation data may be added to thepeak list (generated in step 53) or the match list (generated in step54′), or it may be provided in a separate data structure for access bythe updating step 56. The validation data may take different forms. In afirst variant, the validation data is an indicator (flag, marker, value,etc) associated with a peak to indicate that the peak is eitheruncertain or true. In a second variant, step 54″ operates to delete theuncertain peaks from the peak list, whereby the validation data isimplicitly embedded into the peak list. In a third variant, step 54″operates to determine a probability value for the detected peaks, wherethe probability value indicates the likelihood that a particular peak isfalse (or true). The probability value may be represented as a numericalvalue, in any base, or as a logical value. In a fourth variant, thevalidation data is given as an uncertainty region in the offset pattern.The uncertainty region will signal to the updating step 56 that peaks inthis region may be false. The uncertainty region may, but need not,correspond to the above-mentioned search region (R in FIG. 8).

The updating step 56 takes the validation data into account whenupdating the trace history list. In one embodiment, step 63 may bedesigned to refrain from updating an existing trace based on a matchedpeak that is deemed uncertain. In another embodiment, step 64 isdesigned to refrain from adding a new trace to the trace history listbased on a non-matched (positive) peak that is deemed uncertain.

In yet another embodiment, step 63 and/or step 64 may be designed totentatively track such uncertain peaks for a number of subsequentiterations, to postpone the decision to update the existing trace andadd the new trace, respectively. Such tentative traces may be trackedseparately from the trace history list. Alternatively, they may beincluded in the trace history list, but designated as tentative tracepaths and tentative new traces, respectively. A trace validation step(not shown) may be implemented to take the decision to update when it isverified that the tentatively tracked peak actually forms a trace overthe subsequent iterations, or that the tentative trace has a sufficientextent on the touch surface. The trace validation step may also beimplemented to decide if such a verified tentative trace should beconnected to an existing trace in the trace history list or be added asa new trace. The trace validation step may also delete other tentativetraces from the trace history list.

In a variant of the method in FIG. 5A, step 54″ is executed before thematching step 54′ in the validation step 54. The above description ofsteps 50-57 is equally applicable to such a variant. However, it isconceivable that the matching step 54′ is configured to use thevalidation data to improve the matching process, e.g. to factor in theprobability values/uncertainty region when calculating the match values.

It is realized that the scatter heuristic will reduce the likelihoodthat peaks caused by scatter phenomena are identified as indicative oftouching objects of the touch surface of an FTIR-based system.Furthermore, the scatter heuristic may serve to suppress the influenceof peaks that originate from other types of interference.

5.2 False Peaks Caused by Contamination Changes

When an object is moving along a trajectory on the touch surface, it maychange the contamination on the touch surface. For example, if theobject is a finger, it may both add finger grease to the touch surfaceand rearrange existing contamination on the touch surface. Suchcontamination changes or “residue changes” may be interpreted as peaksin the pattern, even if the pattern generation step 52 is implemented toactively compensate for contaminations, or otherwise suppress theinfluence of contaminations.

FIG. 9A is a schematic view of an object 7 on a touch surface 4, wherethe object has left a trail 90 of contamination as indicated by dashedlines. The attenuation caused by the trail 90 may be estimated andcompensated for by existing techniques. However, it may be difficult tofully compensate for the influence of the contaminations, especially ifthe contaminations are unevenly distributed along and within trail 90.As an example, FIG. 9A indicates two areas 91, 92 in which thecontamination is significantly more than expected. Thereby, the areas91, 92 will result in positive peaks along a movement trajectory in thepattern. This is further illustrated in the offset pattern o_(t) of FIG.10, in which two positive non-matched peaks caused by locally increasedcontamination are observed along an existing trace 30 for a singletouching object. Similar interference may occur in a difference patternda_(t), as illustrated in FIG. 11, where trace 30 contains a positivepeak at the front end of trace 30 caused by the movement of the objectinto a new area on the touch surface during Δt, a negative peak iscaused by the movement of the object away from its previous location,and two positive peaks caused by newly added fingerprints that lead tolocally increased attenuation.

FIG. 9B is a schematic view of an object 7 on a touch surface 4 whichhas an old trail 93 of contamination thereon. The object 7 has beenmoved on the surface 4 while wiping away some of the contamination inthe trail 93, while depositing a new trail 90 of contamination on thesurface 4. If the contamination in trail 93 is not properly estimatedwhen the pattern is generated (step 52 in FIG. 5A), the amount ofcontamination may be either underestimated or overestimated in area 94,resulting in a positive or negative peak along the trace of the object.

To reduce the likelihood that contamination-induced positive peaks areentered into the trace history list as new traces in the updating step56, step 54″ may apply a “contamination heuristic” including a thirdrule that stipulates that a positive peak is to be designated asuncertain if it is located along an existing trace. The contaminationheuristic may contain a fourth rule that stipulates that such a positivepeak is to be designated as uncertain only if it is non-matched toexisting traces (cf. step 54′). The fourth rule will allow the method totrack a touch that crosses another trace, since matched peaks are notidentified by the contamination heuristic.

It is conceivable that the contamination heuristic also includes a fifthrule that stipulates that the positive (matched or non-matched) peak isto be designated as an uncertain peak only if it is located within amaximum distance from the front end of the existing trace. The fifthrule may reduce the risk that new touches that occur in existing tracesare designated as uncertain peaks and prevented from being added to thetrace history list in step 56. Since contamination-induced peaks arelikely to occur relatively close to the front end of the trace, thefifth rule may be used without introducing false peaks into the tracehistory list. The maximum distance used in the fifth rule may be aEuclidian distance or a distance along the trace. In one implementation,the maximum distance is a fixed, absolute distance. In anotherimplementation, the maximum distance is set dynamically as a function ofthe trace velocity, such that a higher trace velocity results in alarger maximum distance. This will ensure that the contamination-inducedpeaks are designated as uncertain also for objects move quickly acrossthe touch surface. In one example, the maximum distance is an extentalong the trace, from its current position to its previous position agiven number of frames back in time, e.g. 5, 10 or 20 frames. Thereby,the positive peak will be designated as uncertain only if it fallswithin the recent trajectory of the trace, where the extent of therecent trajectory automatically scales with the velocity of the trace.

The validation data may be given in any of the above-discussed formats,and may be used as described for scattering-induced peaks. If thevalidation step 54″ is implemented to precede the matching step 54′,step 54″ may be designed to omit the fourth rule (and possibly the fifthrule), whereby all positive peaks that fall on existing traces aredesignated as uncertain peaks. Optionally, the fourth rule (and possiblythe fifth rule) may be applied in the updating step 56, which may bedesigned to include such an uncertain peak in the trace history list ifthe uncertain peak has been reliably matched to an existing trace in thematching step 54′.

While the invention has been described in connection with what ispresently considered to be the most practical and preferred embodiments,it is to be understood that the invention is not limited to thedisclosed embodiments, but on the contrary, is intended to cover variousmodifications and equivalent arrangements included within the spirit andthe scope of the appended claims.

For example, the format of the input values s_(t) depends on (or isgiven by) the reconstruction function

′ and the desired/required accuracy of the reconstructed pattern. Inalternative implementations, the format may e.g. be given as atransmission (e.g. given by the projection value normalized by thereference value), an energy difference (e.g. given by the differencebetween the projection value and the reference value), or a logarithm ofthe energy difference. As used hereinabove, a “logarithm” is intended toalso encompass functions approximating a true logarithmic function, inany base. Furthermore, the input values may have any sign, i.e. they mayor may not be multiplied by a negative value. Depending on thereconstruction function, it may even be possible to use the projectionvalues as input values. Similar modifications may be made with respectto the generation of the difference pattern. Thus, on a more generallevel, the above-mentioned total, offset and difference patterns may beregarded as “interaction patterns” that comprise a distribution ofchanges in local interaction with propagating light caused by one ormore touching objects.

In one category of touch-sensitive panels known as ‘above surfaceoptical touch systems,’ an example of which is shown in U.S. Pat. No.4,459,476, a plurality of optical emitters and optical receivers arearranged around the periphery of a touch surface to create a grid ofintersecting light paths above the touch surface. Each light pathextends between a respective emitter/receiver pair. An object thattouches the touch surface blocks certain ones of the light paths. Basedon the identity of the receivers detecting a blocked light path, aprocessor may determine the location of the intercept between theblocked light paths. Conventionally, this type of system is only capableof detecting the location of one object (single-touch detection).Further, the required number of emitters and receivers, and thus costand complexity, increases rapidly with increasing surface area and/orspatial resolution of the touch panel. In an example variant shown inInternational Application Publication No. WO 2006/095320, each opticalemitter emits a beam of light that diverges across the touch surface,and each beam is detected by more than one optical receiver positionedaround the periphery of the touch surface. Thus, each emitter createsmore than one light path across the touch surface. In this example, arelatively large number of light paths are created by sequentiallyactivating different emitters around the periphery of the touch surface,and detecting the light received from each emitter by a plurality ofoptical receivers. Thereby, the number of emitters and receivers for agiven surface area or spatial resolution may be reduced and/orsimultaneous detection of locations of more than one touching object(multi-touch detection) may be enabled.

In at least one example embodiment of the touch-sensitive apparatus 100shown in FIGS. 1A and 1B, an ‘above surface’ arrangement may be used. Inat least this example, light propagates from emitters 2 to detectors 3above the surface of the panel, rather than within the panel via FTIR.In such an example embodiment, whereas objects touching the panel maycompletely block the light paths, artifacts of the reconstructionprocess may cause negative and positive apparent peaks. Tomographicreconstruction algorithms used for transmission tomography use a modelfor touch attenuation that assumes some transmission of light throughevery point in the active area, even in the presence of a touchingobject. In a blocking path touch system, light is blocked completely andthe model of touch attenuation used by transmission tomography may beinadequate.

If an elongated touch object (e.g., a palm or an eraser) is considered,the ordinary mathematical model for transmission tomography may notaccurately represent the actual physical model. However, since acomplete modelling of the physical model is relatively expensive atleast computationally, at least one example embodiment uses a standardtransmission tomography algorithm despite the limitations, and in whichthe resulting errors are handled using the above described techniques.In at least one example embodiment, the tomographic reconstructionprocess finds the minimal 2-norm solution (sum of least square errors)to the simplified mathematical model of the system. The mathematicalalgorithm produces negative and positive apparent peaks outside of theshort sides of the touch object, and these peaks may be tracked byutilizing methods according to one or more example embodiments discussedherein.

The foregoing description has been provided for purposes of illustrationand description. It is not intended to be exhaustive or to limit thedisclosure. Individual elements or features of a particular exampleembodiment are generally not limited to that particular embodiment, but,where applicable, are interchangeable and can be used in a selectedembodiment, even if not specifically shown or described. The same mayalso be varied in many ways. Such variations are not to be regarded as adeparture from the disclosure, and all such modifications are intendedto be included within the scope of the disclosure.

The invention claimed is:
 1. A method of tracking movement of one ormore objects on a touch surface of a touch-sensitive apparatus, thetouch-sensitive apparatus being operable to propagate light from aplurality of incoupling points to a plurality of outcoupling points,wherein at least one light detector is optically coupled to theplurality of outcoupling points to generate an output signalrepresenting received light at the plurality of outcoupling points, saidmethod comprising: processing the output signal to generate aninteraction map indicative of local changes in interaction on the touchsurface; processing the interaction map to identify positive andnegative apparent peaks, each positive apparent peak representing alocally increased interaction in the interaction map, and each negativeapparent peak representing a locally decreased interaction in theinteraction map; processing the positive and negative apparent peaks toidentify potentially false peaks among the positive and negativeapparent peaks; and updating, based on the potentially false peaks,movement trajectories of all objects deemed to exist on the touchsurface at a preceding time point, wherein the processing the positiveand negative apparent peaks applies at least a first heuristic toidentify the potentially false peaks, said first heuristic designating apositive apparent peak as a potentially false peak when the positiveapparent peak is deemed to be associated with one or more of thenegative apparent peaks.
 2. The method of claim 1, wherein said one ormore of the negative apparent peaks are located outside the movementtrajectories.
 3. The method of claim 1, wherein the positive apparentpeak is deemed to be associated with said one or more of the negativeapparent peaks when the positive apparent peak has a given spatialrelation to said one or more of the negative apparent peaks.
 4. Themethod of claim 3, wherein the given spatial relation defines a maximumdistance between the positive apparent peak and said one or more of thenegative apparent peaks.
 5. The method of claim 1, wherein theprocessing the positive and negative apparent peaks comprises: matchingat least part of the positive and negative apparent peaks to themovement trajectories to identify a set of matched apparent peaks and aset of non-matched apparent peaks.
 6. The method of claim 1, wherein theupdating comprises: evaluating, based on the potentially false peaks,the positive and negative apparent peaks to determine a current locationfor at least part of the movement trajectories; evaluating, based on thepotentially false peaks, the positive and negative apparent peaks toidentify new touches; and adding a location of each of said new touchesas a starting point of a new movement trajectory.
 7. The method of claim6, wherein the updating further comprises: preventing identification ofnew touches based on the potentially false peaks.
 8. The method of claim7, wherein the method is successively repeated; and the identificationof new touches based on the potentially false peaks is prevented duringa number of successive repetitions for at least part of the potentiallyfalse peaks.
 9. The method of claim 6, wherein the updating furthercomprises at least one of: updating the movement trajectories based onat least part of the positive and negative apparent peaks while omittingall potentially false peaks; updating the movement trajectories based onsaid at least part of the positive and negative apparent peaks whileapplying a respective peak probability value indicative of thepotentially false peaks; postponing updating of at least a subset ofmovement trajectories that are deemed to be associated with thepotentially false peaks; creating tentative movement trajectories for atleast a subset of the potentially false peaks; and postponing theupdating of movement trajectories that are deemed to be associated withone or more uncertainty regions containing the potentially false peaksin the interaction map.
 10. The method of claim 1, further comprising:identifying current touch data for at least part of the updated movementtrajectories; and outputting the current touch data.
 11. The method ofclaim 1, wherein the processing the output signal comprises: processingthe output signal to identify signal values for a plurality of lightpaths extending between pairs of the plurality of incoupling and theplurality of outcoupling points; and generating the interaction map byoperating an image reconstruction algorithm on the signal values. 12.The method of claim 1, wherein the interaction is represented as one ofattenuation and transmission of said light.
 13. The method of claim 1,further comprising: retrieving, from an electronic memory, a datastructure indicative of the movement trajectories of all objects deemedto exist on the touch surface at the preceding time point; and whereinthe updating includes updating the movement trajectories in the datastructure; and storing the updated data structure in the electronicmemory.
 14. A non-transitory computer-readable storage medium storingcomputer executable instructions that, when executed on adata-processing system, cause the data-processing system to carry outthe method of claim
 1. 15. A device for tracking movement of one or moreobjects on a touch surface of a touch-sensitive apparatus, thetouch-sensitive apparatus being operable to propagate light from aplurality of incoupling points to a plurality of outcoupling points,wherein a light sensor is optically coupled to the plurality ofoutcoupling points to generate an output signal representing receivedlight at the plurality of outcoupling points, said device comprising:means for processing the output signal to generate an interaction mapindicative of local changes in interaction on the touch surface; meansfor processing the interaction map to identify positive and negativeapparent peaks, each positive apparent peak representing a locallyincreased interaction in the interaction map, and each negative apparentpeak representing a locally decreased interaction in the interactionmap; means for processing the positive and negative apparent peaks toidentify potentially false peaks among the positive and negativeapparent peaks; and means for updating, based on the potentially falsepeaks, movement trajectories of all objects deemed to exist on the touchsurface at a preceding time point, wherein the means for processing thepositive and negative apparent peaks applies at least a first heuristicto identify the potentially false peaks, said first heuristicdesignating a positive apparent peak as a potentially false peak whenthe positive apparent peak is deemed to be associated with one or moreof the negative apparent peaks.
 16. A device for tracking movement ofone or more objects on a touch surface of a touch-sensitive apparatus,the touch-sensitive apparatus being operable to propagate light from aplurality of incoupling points to a plurality of outcoupling points soas to illuminate the touch surface, wherein a light sensor is opticallycoupled to the plurality of outcoupling points to generate an outputsignal representing received light at the plurality of outcouplingpoints, said device comprising: at least one processor configured toexecute computer-readable instructions to process the output signal togenerate an interaction map indicative of local changes in interactionon the touch surface; process the interaction map to identify positiveand negative apparent peaks, each positive apparent peak representing alocally increased interaction in the interaction map, and each negativeapparent peak representing a locally decreased interaction in theinteraction map; process the positive and negative apparent peaks toidentify potentially false peaks among the positive and negativeapparent peaks by applying at least a first heuristic to identify thepotentially false peaks, said first heuristic designating a positiveapparent peak as a potentially false peak when the positive apparentpeak is deemed to be associated with one or more of the negativeapparent peaks; and update, based on the potentially false peaks,movement trajectories of all objects deemed to exist on the touchsurface at a preceding time point.
 17. The device of claim 16, whereinsaid one or more of the negative apparent peaks are located outside themovement trajectories.
 18. The device of claim 16, wherein the positiveapparent peak is deemed to be associated with said one or more of thenegative apparent peaks when the positive apparent peak has a givenspatial relation to said one or more of the negative apparent peaks. 19.The device of claim 18, wherein the given spatial relation defines amaximum distance between the positive apparent peak and said one or moreof the negative apparent peaks.
 20. The device of claim 16, wherein theat least one processor is further configured to executecomputer-readable instructions to match at least part of the positiveand negative apparent peaks to the movement trajectories to identify aset of matched apparent peaks and a set of non-matched apparent peaks.